K-EmoPhone: A Mobile and Wearable Dataset with In-Situ Emotion, Stress, and Attention Labels.

With the popularization of low-cost mobile and wearable sensors, several studies have used them to track and analyze mental well-being, productivity, and behavioral patterns. However, there is still a lack of open datasets collected in real-world contexts with affective and cognitive state labels such as emotion, stress, and attention; the lack of such datasets limits research advances in affective computing and human-computer interaction. This study presents K-EmoPhone, a real-world multimodal dataset collected from 77 students over seven days. This dataset contains (1) continuous probing of peripheral physiological signals and mobility data measured by commercial off-the-shelf devices, (2) context and interaction data collected from individuals' smartphones, and (3) 5,582 self-reported affect states, including emotions, stress, attention, and task disturbance, acquired by the experience sampling method. We anticipate the dataset will contribute to advancements in affective computing, emotion intelligence technologies, and attention management based on mobile and wearable sensor data.

Loss-Framed Adaptive Microcontingency Management for Preventing Prolonged Sedentariness: Development and Feasibility Study.

BACKGROUND: A growing body of evidence shows that financial incentives can effectively reinforce individuals' positive behavior change and improve compliance with health intervention programs. A critical factor in the design of incentive-based interventions is to set a proper incentive magnitude. However, it is highly challenging to determine such magnitudes as the effects of incentive magnitude depend on personal attitudes and contexts. OBJECTIVE: This study aimed to illustrate loss-framed adaptive microcontingency management (L-AMCM) and the lessons learned from a feasibility study. L-AMCM discourages an individual's adverse health behaviors by deducting particular expenses from a regularly assigned budget, where expenses are adaptively estimated based on the individual's previous responses to varying expenses and contexts. METHODS: We developed a mobile health intervention app for preventing prolonged sedentary lifestyles. This app delivered a behavioral mission (ie, suggesting taking an active break for a while) with an incentive bid when 50 minutes of uninterrupted sedentary behavior happened. Participants were assigned to either the fixed (ie, deducting the monotonous expense for each mission failure) or adaptive (ie, deducting varying expenses estimated by the L-AMCM for each mission failure) incentive group. The intervention lasted 3 weeks. RESULTS: We recruited 41 participants (n=15, 37% women; fixed incentive group: n=20, 49% of participants; adaptive incentive group: n=21, 51% of participants) whose mean age was 24.0 (SD 3.8; range 19-34) years. Mission success rates did not show statistically significant differences by group (P=.54; fixed incentive group mean 0.66, SD 0.24; adaptive incentive group mean 0.61, SD 0.22). The follow-up analysis of the adaptive incentive group revealed that the influence of incentive magnitudes on mission success was not statistically significant (P=.18; odds ratio 0.98, 95% CI 0.95-1.01). On the basis of the qualitative interviews, such results were possibly because the participants had sufficient intrinsic motivation and less sensitivity to incentive magnitudes. CONCLUSIONS: Although our L-AMCM did not significantly affect users' mission success rate, this study configures a pioneering work toward adaptively estimating incentives by considering user behaviors and contexts through leveraging mobile sensing and machine learning. We hope that this study inspires researchers to develop incentive-based interventions.

LSTM-Modeling of Emotion Recognition Using Peripheral Physiological Signals in Naturalistic Conversations.

The automated recognition of human emotions plays an important role in developing machines with emotional intelligence. Major research efforts are dedicated to the development of emotion recognition methods. However, most of the affective computing models are based on images, audio, videos and brain signals. Literature lacks works that focus on utilizing only peripheral signals for emotion recognition (ER), which can be ideally implemented in daily life settings. Therefore, this paper present a framework for ER on the arousal and valence space, based on using multi-modal peripheral signals. The data used in this work were collected during a debate between two people using wearable devices. The emotions of the participants were rated by multiple raters and converted into classes in correspondence to the arousal and valence space. The use of a dynamic threshold for ratings conversion was investigated. An ER model is proposed that uses a Long Short-Term Memory (LSTM)-based architecture for classification. The model uses heart rate (HR), temperature (T), and electrodermal activity (EDA) signals as its inputs with emotional cues. Additionally, a post-processing prediction mechanism is introduced to enhance the recognition performance. The model is implemented to study the use of individual and different combinations of the peripheral signals, as well as utilizing annotations from different ratings. Additionally, it is employed for classification of valence and arousal in an independent and combined fashion, under subject dependent and independent scenarios. The experimental results have justified the efficient performance of the proposed framework, achieving classification accuracy 96% and 93% for the independent and combined classification scenarios, accordingly. The comparison of the achieved performance against the baseline methods shows the superiority of the proposed framework and the ability to recognize arousal-valance levels with high accuracy from peripheral signals, in real-life scenarios.

Trends in Smart Helmets With Multimodal Sensing for Health and Safety: Scoping Review.

BACKGROUND: As a form of the Internet of Things (IoT)-gateways, a smart helmet is one of the core devices that offers distinct functionalities. The development of smart helmets connected to IoT infrastructure helps promote connected health and safety in various fields. In this regard, we present a comprehensive analysis of smart helmet technology and its main characteristics and applications for health and safety. OBJECTIVE: This paper reviews the trends in smart helmet technology and provides an overview of the current and future potential deployments of such technology, the development of smart helmets for continuous monitoring of the health status of users, and the surrounding environmental conditions. The research questions were as follows: What are the main purposes and domains of smart helmets for health and safety? How have researchers realized key features and with what types of sensors? METHODS: We selected studies cited in electronic databases such as Google Scholar, Web of Science, ScienceDirect, and EBSCO on smart helmets through a keyword search from January 2010 to December 2021. In total, 1268 papers were identified (Web of Science: 87/1268, 6.86%; EBSCO: 149/1268, 11.75%; ScienceDirect: 248/1268, 19.55%; and Google Scholar: 784/1268, 61.82%), and the number of final studies included after PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) study selection was 57. We also performed a self-assessment of the reviewed articles to determine the quality of the paper. The scoring was based on five criteria: test environment, prototype quality, feasibility test, sensor calibration, and versatility. RESULTS: Smart helmet research has been considered in industry, sports, first responder, and health tracking scenarios for health and safety purposes. Among 57 studies, most studies with prototype development were industrial applications (18/57, 32%), and the 2 most frequent studies including simulation were industry (23/57, 40%) and sports (23/57, 40%) applications. From our assessment-scoring result, studies tended to focus on sensor calibration results (2.3 out of 3), while the lowest part was a feasibility test (1.6 out of 3). Further classification of the purpose of smart helmets yielded 4 major categories, including activity, physiological and environmental (hazard) risk sensing, as well as risk event alerting. CONCLUSIONS: A summary of existing smart helmet systems is presented with a review of the sensor features used in the prototyping demonstrations. Overall, we aimed to explore new possibilities by examining the latest research, sensor technologies, and application platform perspectives for smart helmets as promising wearable devices. The barriers to users, challenges in the development of smart helmets, and future opportunities for health and safety applications are also discussed. In conclusion, this paper presents the current status of smart helmet technology, main issues, and prospects for future smart helmet with the objective of making the smart helmet concept a reality.

Beyond Pathogen Filtration: Possibility of Smart Masks as Wearable Devices for Personal and Group Health and Safety Management.

Face masks are an important way to combat the COVID-19 pandemic. However, the prolonged pandemic has revealed confounding problems with the current face masks, including not only the spread of the disease but also concurrent psychological, social, and economic complications. As face masks have been worn for a long time, people have been interested in expanding the purpose of masks from protection to comfort and health, leading to the release of various "smart" mask products around the world. To envision how the smart masks will be extended, this paper reviewed 25 smart masks (12 from commercial products and 13 from academic prototypes) that emerged after the pandemic. While most smart masks presented in the market focus on resolving problems with user breathing discomfort, which arise from prolonged use, academic prototypes were designed for not only sensing COVID-19 but also general health monitoring aspects. Further, we investigated several specific sensors that can be incorporated into the mask for expanding biophysical features. On a larger scale, we discussed the architecture and possible applications with the help of connected smart masks. Namely, beyond a personal sensing application, a group or community sensing application may share an aggregate version of information with the broader population. In addition, this kind of collaborative sensing will also address the challenges of individual sensing, such as reliability and coverage. Lastly, we identified possible service application fields and further considerations for actual use. Along with daily-life health monitoring, smart masks may function as a general respiratory health tool for sports training, in an emergency room or ambulatory setting, as protection for industry workers and firefighters, and for soldier safety and survivability. For further considerations, we investigated design aspects in terms of sensor reliability and reproducibility, ergonomic design for user acceptance, and privacy-aware data-handling. Overall, we aim to explore new possibilities by examining the latest research, sensor technologies, and application platform perspectives for smart masks as one of the promising wearable devices. By integrating biomarkers of respiration symptoms, a smart mask can be a truly cutting-edge device that expands further knowledge on health monitoring to reach the next level of wearables.

Successfully Implementing Digital Health to Ensure Future Global Health Security During Pandemics: A Consensus Statement.

Importance: COVID-19 has highlighted widespread chronic underinvestment in digital health that hampered public health responses to the pandemic. Recognizing this, the Riyadh Declaration on Digital Health, formulated by an international interdisciplinary team of medical, academic, and industry experts at the Riyadh Global Digital Health Summit in August 2020, provided a set of digital health recommendations for the global health community to address the challenges of current and future pandemics. However, guidance is needed on how to implement these recommendations in practice. Objective: To develop guidance for stakeholders on how best to deploy digital health and data and support public health in an integrated manner to overcome the COVID-19 pandemic and future pandemics. Evidence Review: Themes were determined by first reviewing the literature and Riyadh Global Digital Health Summit conference proceedings, with experts independently contributing ideas. Then, 2 rounds of review were conducted until all experts agreed on the themes and main issues arising using a nominal group technique to reach consensus. Prioritization was based on how useful the consensus recommendation might be to a policy maker. Findings: A diverse stakeholder group of 13 leaders in the fields of public health, digital health, and health care were engaged to reach a consensus on how to implement digital health recommendations to address the challenges of current and future pandemics. Participants reached a consensus on high-priority issues identified within 5 themes: team, transparency and trust, technology, techquity (the strategic development and deployment of technology in health care and health to achieve health equity), and transformation. Each theme contains concrete points of consensus to guide the local, national, and international adoption of digital health to address challenges of current and future pandemics. Conclusions and Relevance: The consensus points described for these themes provide a roadmap for the implementation of digital health policy by all stakeholders, including governments. Implementation of these recommendations could have a significant impact by reducing fatalities and uniting countries on current and future battles against pandemics.

Arousal-Valence Classification from Peripheral Physiological Signals Using Long Short-Term Memory Networks.

The automated recognition of human emotions plays an important role in developing machines with emotional intelligence. However, most of the affective computing models are based on images, audio, videos and brain signals. There is a lack of prior studies that focus on utilizing only peripheral physiological signals for emotion recognition, which can ideally be implemented in daily life settings using wearables, e.g., smartwatches. Here, an emotion classification method using peripheral physiological signals, obtained by wearable devices that enable continuous monitoring of emotional states, is presented. A Long Short-Term Memory neural network-based classification model is proposed to accurately predict emotions in real-time into binary levels and quadrants of the arousal-valence space. The peripheral sensored data used here were collected from 20 participants, who engaged in a naturalistic debate. Different annotation schemes were adopted and their impact on the classification performance was explored. Evaluation results demonstrate the capability of our method with a measured accuracy of >93% and >89% for binary levels and quad classes, respectively. This paves the way for enhancing the role of wearable devices in emotional state recognition in everyday life.

Validation of Self-Reported Smartphone Usage Against Objectively-Measured Smartphone Usage in Hong Kong Chinese Adolescents and Young Adults.

OBJECTIVE: This study evaluated the validity of self-reported smartphone usage data against objectively-measured smartphone usage data by directly tracking the activities in the participants' smartphone among Chinese adolescents and young adults in Hong Kong. METHODS: A total of 187 participants were recruited (mean age 19.4, 71.7% female) between 2017 and 2018. A smartphone usage tracking app was installed on all participants' smartphone for 7 consecutive days. After the 7-day monitoring period, they completed a selfadministered questionnaire on smartphone usage habits. RESULTS: Although the correlation between self-reported and objectively-measured total smartphone usage time was insignificant (ρ=-0.10, p=0.18), in three out of the four usage domains were positively and significantly correlated, namely social network (ρ=0.21, p=0.005), instant messaging (ρ=0.27, p<0.001), and games (ρ=0.64, p<0.001). Participants' self-report of the total time spent on smartphones exceeded the objective data by around 760 min per week (self-reported 1,930.3 min/wk vs. objectively-measured 1,170.7 min/wk, p<0.001). Most of the over-reporting was contributed by the web browsing domain (self-reported 447.8 min/wk vs. objectively-measured 33.3 min/wk, p<0.001). CONCLUSION: Our results showed large discrepancies between self-reported smartphone and objectively-measured smartphone usage except for self-reported usage on game apps.

Temporal association between objectively measured smartphone usage, sleep quality and physical activity among Chinese adolescents and young adults.

We studied the association between objectively measured smartphone usage and objectively measured sleep quality and physical activity for seven consecutive days among Hong Kong adolescents and young adults aged 11-25 years (n = 357, 67% female). We installed an app that tracked the subjects' smartphone usage and had them wear an ActiGraph GT3X accelerometer on their wrist to measure their sleep quality and physical activity level. Smartphone usage data were successfully obtained from 187 participants (52.4%). The participants on average spent 2 h 46 min per day on their smartphone. Multilevel regression showed that 1 min of daytime smartphone usage was associated with 0.07 min decrease in total sleeping time that night (p = .043, 95% confidence interval [CI]: -0.14, -0.003). Broken down for different usage purposes, 1 min of daytime social network usage and games and comics was associated with a 0.28 (p = .02, 95% CI: -0.52, -0.04) min and 0.18 min (p = .01, 95% CI: -0.32, -0.04) decrease in total sleeping time that night, respectively. One minute of daytime smartphone usage was associated with an increase of 4.55 steps in the number of steps (p = .001, 95% CI: 1.77, 7.34) on the next day. To conclude, time spent on a smartphone in the daytime was associated with total sleeping time that night and number of steps the next day, but was not associated with sleep efficiency, wake after sleep onset and moderate-to-vigorous-intensity activity (MVPA) among Hong Kong adolescents and young adults.

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations.

Recognizing emotions during social interactions has many potential applications with the popularization of low-cost mobile sensors, but a challenge remains with the lack of naturalistic affective interaction data. Most existing emotion datasets do not support studying idiosyncratic emotions arising in the wild as they were collected in constrained environments. Therefore, studying emotions in the context of social interactions requires a novel dataset, and K-EmoCon is such a multimodal dataset with comprehensive annotations of continuous emotions during naturalistic conversations. The dataset contains multimodal measurements, including audiovisual recordings, EEG, and peripheral physiological signals, acquired with off-the-shelf devices from 16 sessions of approximately 10-minute long paired debates on a social issue. Distinct from previous datasets, it includes emotion annotations from all three available perspectives: self, debate partner, and external observers. Raters annotated emotional displays at intervals of every 5 seconds while viewing the debate footage, in terms of arousal-valence and 18 additional categorical emotions. The resulting K-EmoCon is the first publicly available emotion dataset accommodating the multiperspective assessment of emotions during social interactions.

Too Much Information: Assessing Privacy Risks of Contact Trace Data Disclosure on People With COVID-19 in South Korea.

Introduction: With the COVID-19 outbreak, South Korea has been making contact trace data public to help people self-check if they have been in contact with a person infected with the coronavirus. Despite its benefits in suppressing the spread of the virus, publicizing contact trace data raises concerns about individuals' privacy. In view of this tug-of-war between one's privacy and public safety, this work aims to deepen the understanding of privacy risks of contact trace data disclosure practices in South Korea. Method: In this study, publicly available contact trace data of 970 confirmed patients were collected from seven metropolitan cities in South Korea (20th Jan-20th Apr 2020). Then, an ordinal scale of relative privacy risk levels was introduced for evaluation, and the assessment was performed on the personal information included in the contact trace data, such as demographics, significant places, sensitive information, social relationships, and routine behaviors. In addition, variance of privacy risk levels was examined across regions and over time to check for differences in policy implementation. Results: It was found that most of the contact trace data showed the gender and age of the patients. In addition, it disclosed significant places (home/work) ranging across different levels of privacy risks in over 70% of the cases. Inference on sensitive information (hobby, religion) was made possible, and 48.7% of the cases exposed the patient's social relationships. In terms of regional differences, a considerable discrepancy was found in the privacy risk for each category. Despite the recent release of government guidelines on data disclosure, its effects were still limited to a few factors (e.g., workplaces, routine behaviors). Discussion: Privacy risk assessment showed evidence of superfluous information disclosure in the current practice. This study discusses the role of "identifiability" in contact tracing to provide new directions for minimizing disclosure of privacy infringing information. Analysis of real-world data can offer potential stakeholders, such as researchers, service developers, and government officials with practical protocols/guidelines in publicizing information of patients and design implications for future systems (e.g., automatic privacy sensitivity checking) to strike a balance between one's privacy and the public benefits with data disclosure.

Cognitive States Matter: Design Guidelines for Driving Situation Awareness in Smart Vehicles.

Situation awareness (SA) is crucial for safe driving. It is all about perception, comprehension of current situations and projection of the future status. It is demanding for drivers to constantly maintain SA by checking for potential hazards while performing the primary driving tasks. As vehicles in the future will be equipped with more sensors, it is likely that an SA aiding system will present complex situational information to drivers. Although drivers have difficulty to process a variety of complex situational information due to limited cognitive capabilities and perceive the information differently depending upon their cognitive states, the well-known SA design principles by Endsley only provide general guidelines. The principles lack detailed guidelines for dealing with limited human cognitive capabilities. Cognitive capability is a mental capability including planning, complex idea comprehension, and learning from experience. A cognitive state can be regarded as a condition of being (e.g., the state of being aware of the situation). In this paper, we investigate the key cognitive attributes related to SA in driving contexts (i.e., attention focus, mental model, workload, and memory). Endsley proposed that those key cognitive attributes are the main factors that influence SA. In those with higher levels of attributes, we found eight cognitive states which mainly influence a human driver in achieving SA. These are the focused attention state, inattentional blindness state, unfamiliar situation state, familiar situation state, insufficient mental resource state, sufficient mental resource state, high time pressure state, and low time pressure state. We then propose cognitive state aware SA design guidelines that can help designers to effectively convey situation information to drivers. As a case study, we demonstrated the usefulness of our cognitive state aware SA design guidelines by conducting controlled experiments where an existing SA interface is compared with a new SA interface designed following the key guidelines. We used the Situation Awareness Global Assessment Technique (SAGAT) and Decision-Making Questionnaire (DMQ) to measure the SA and decision-making style scores, respectively. Our results show that the new guidelines allowed participants to achieve significantly higher SA and exhibit better decision making performance.

What Makes Sports Fans Interactive? Identifying Factors Affecting Chat Interactions in Online Sports Viewing.

Sports fans are able to watch games from many locations using TV services while interacting with other fans online. In this paper, we identify the factors that affect sports viewers' online interactions. Using a large-scale dataset of more than 25 million chat messages from a popular social TV site for baseball, we extract various game-related factors, and investigate the relationships between these factors and fans' interactions using a series of multiple regression analyses. As a result, we identify several factors that are significantly related to viewer interactions. In addition, we determine that the influence of these factors varies according to the user group; i.e., active vs. less active users, and loyal vs. non-loyal users.